The term “search engine optimization” (SEO) is broadly applied to any of several techniques used by content providers to enhance the visibility of their content (e.g., web sites and the like) in a search engine's natural or unpaid search results. It is generally agreed that results which appear closer to the top (or earlier) in a list of ranked search results will be more likely to be frequented by visitors than will results which appear lower (or later) in such a list. Consequently, content providers employ SEO methodologies in an effort to have their content appear as early as possible in such lists.
The effectiveness of any one or more SEO techniques depends on a variety of factors, including how different search engines index and/or catalog the subject content, the number of backlinks to the subject content, and how users of the search engines employ those tools. Thus, SEO techniques that may be effective for some search engines may not be as effective for others. Regardless of the particular techniques, however, all SEO methodologies are premised on the assumption that the subject content is visible to the search engine indexing processes and to users that employ the search engines and click through to that content when it is presented in the search results.
This requirement of visibility presents a problem for many content owners. For example, allowing “free” access to the content items (e.g., via search engine result-based links) would be problematic (perhaps antithetical) to any business model that depended on license fees or similar bases for granting access (generally referred to herein as “paid-access”) to content items. Some content owners deal with this dilemma by making portions of individual content items freely available. For example, in the case of journal articles and the like, abstracts or similar summaries are often freely available and so can be indexed by search engines. As long as the abstract accurately captures the true nature of the underlying article this may be sufficient to permit the article to be included in search results, but it is unlikely that the abstract can be optimized in a way that it would enhance its position within such results. Therefore, the usefulness of such techniques as ways for content owners to attract potential new subscribers or purchasers is questionable.
The vast amount of content hidden behind subscription, registration or similar barriers is part of what is generally referred to as the “deep web” or “invisible web”, referring to the fact that it is a collection of material that most Internet users are unaware of because it is never indexed by search engines. Recognizing that allowing users access to invisible web content will likely enhance the user's search experience, some search engine providers have implemented programs to allow content providers to have their paid-access or registration-required content indexed by their search engines. Under these programs, content providers must generally agree to allow users that find the content items through a search involving the subject search engine to see the full text of the subject document without charge and without need for registration. Subsequent attempts to access content from the content provider's site by clicking links in the original document may be restricted to those users who register and/or pay for such privileges. One such program implemented by Google Inc. is referred to as “First Click Free”. In addition to allowing Google users free access to the full text of any document discovered through a Google search, the content providers must also provide unrestricted access to that document to the Googlebot—Google's web crawling and indexing bot or spider.